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ABSTRACT 


This work presents a discussion about the relationship between the contributions of Alan Turing 
— the centenary of whose birth is celebrated in 2012 — to the field of artificial neural networks 
and modern unorganized machines: reservoir computing (RC) approaches and extreme learning 
machines (ELMs). Firstly, we review Turing’s connectionist proposals and also expose the 
fundamentals of the main RC paradigms — echo state networks and liquid state machines -, as 
well as of the design and training of ELMs. Throughout this exposition, the main points of 
contact between Turing’s ideas and these modern perspectives are outlined, being, then, duly 
summarized in the second and final part of the work. We hope this paper be useful in offering a 
distinct appreciation of Turing’s pioneering contributions to the field of neural networks and 
also in indicating some perspectives for the future development of the field that may arise from 
the synergy between these views. 


Keywords. Neural networks, connectionism, Alan Turing, unorganized machines, machine 
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1. INTRODUCTION 


Alan Mathison Turing (1912-1954) is widely regarded as one of the foremost mathematicians of 
the 20" century (Hawking, 2007), especially due to his proof of the impossibility of existence of 
a general solution to the Entscheidungsproblem posed by David Hilbert (Turing, 1936). Albeit 
Alonzo Church, in a completely independent fashion, has reached the same conclusion using the 
formalism known as A-calculus (Epstein & Carnielli, 2008) — which would play a role of its own 
in theoretical computer science (Russell & Norvig, 2009) -, Turing’s proof was striking in the 
sense that it embodied, in an abstract (though almost tangible) way, the essence of the computing 
machines that were to occupy a preponderant role in the technological history of mankind from 
the 1940s up to present day. A result of this nature has the weight of a lifetime achievement, but, 


for Turing, it was only a sort of “first peak” in an intellectual trajectory that would also leave 
marks in other chapters of the history of the last century. 


In September 1938, the same year in which he concluded his PhD under Church’s 
supervision, Turing reported to Government Code and Cypher School (GCCS) at Bletchley Park 
(Hodges, 1992). Hence began a work on the decryption of the naval Enigma cipher that had a 
remarkable impact on the British war effort, which suffered immensely from the action of the 
German submarines. The concurrent problem of speech coding allowed Turing to have contact 
with the Bell Labs team, which apropos included Claude Shannon, who shared many of Turing’s 
views regarding what can be broadly termed “information science”. In a tour the force involving 
cryptography, early signal processing theory and electronics, Turing conceived a voice-encoding 
device called Delilah (“the biblical deceiver of men”, as explained in (Hodges, 1992)), the 
prototype of which was built in 1944. The system, however, never had the chance of playing the 
practical role for which it was intended. 


In parallel with these developments, Turing cultivated an interest that was central enough 
to leave a definite mark in his post-war research corpus: the idea of building a thinking machine 
(Leavitt, 2006). This idea is organically related to many of his earlier intellectual pursuits’, and 
also to certain philosophical convictions forged in the course of his personal history. 


One particular effort was very influential in the development of the field of artificial 
intelligence (AI) (Russell & Norvig, 2009): the attempt to formalize the notion of machine 
intelligence using an imitation game that is now termed Turing test. The test is discussed in a 
work (Turing, 1950) that is an immense pleasure to read, not only for its unique style, but also 
for its cornucopia of brilliant insights into many aspects and key questions associated with the 
later development of AI. 


Interestingly, Turing also made other important contributions to the field that, in spite of 
their relevance, are, even presently, relatively little known. Among these contributions, we 
highlight, in this paper, Turing’s ideas on the design of neural networks — which, thanks to works 
like (Teuscher, 2001) and (Copeland & Proudfoot, 1999; Copeland, 2004), are starting to receive 
a more fit appreciation. These ideas can be, from a historical standpoint, considered as a 
development of the logical approach to neural modeling introduced by Warren McCulloch and 
Walter Pitts (1943). However, as we will discuss later on, Turing’s connectionist perspective is 
also very rich in its early use of recurrence and, particularly, of unorganized architectures. These 
features establish a number of interesting parallels with some recent neural approaches — like 
reservoir computing (Lukosevicius & Jaeger, 2009) and extreme learning machines (Huang, Zhu, 
& Siew, 2006) - and also with a number of investigations concerning the connectivity pattern of 
the nervous system. These connections have not, to the best of our knowledge, hitherto been 
explored. It is our belief that they can be useful not only in historiographical terms, but also as 
indicative of promising research subjects. 


This work is organized as follows: in Section 2, we review Turing’s seminal ideas 
introduced in the report Intelligent Machinery, focusing on his contributions to the field of neural 
networks through the proposal of the so-called unorganized machines; then, in Sections 3 and 4, 
we describe in detail two modern connectionist paradigms — reservoir computing and extreme 
learning machines — that revive the idea of “disorganization”; the conceptual links between these 
approaches and Turing’s ideas are duly established in Section 5, in which we also discuss 


interesting perspectives for future research that may benefit from this synergy. Finally, Section 6 
concludes the paper. 


2. TURING’S NEURAL NETWORKS 


In 1948, Turing delivered to Sir Charles Darwin, head of the National Physical Laboratory 
(NPL), in London, the report entitled Intelligent Machinery, describing the main results of his 
research concerning the possibility that computational machines exhibit intelligent behavior 
analogous to that engendered by the human brain. In his own words: “I propose to investigate 
the question as to whether it is possible for machinery to show intelligent behaviour.” (Turing, 
1968). Unfortunately, Intelligent Machinery did not receive the deserved appreciation at the 
time, having remained unpublished until 1968, fourteen years after Turing’s death. Recently, due 
to the efforts of Teuscher (2001), Copeland (2004) and others, this landmark work has been 
progressively disseminated and its valuable contributions are finally being properly 
acknowledged. 


In this remarkably original report, Turing anticipated several key concepts that would 
later be central to the field of artificial / computational intelligence. For instance, the use of 
search mechanisms inspired in evolution theory, such as genetic algorithms (Holland, 1975) and 
evolution strategies (Schwefel, 1981), can be found in nuce in what he called “evolutionary or 
genetic search” (Turing, 1968). Nonetheless, Intelligent Machinery is mainly devoted to a 
discussion about machine learning, in which Turing anticipates the approach modernly known as 
connectionism (Turing, 1968). 


It is important to stress that the idea of using a network composed of interconnected 
simple processing units had already been addressed in the work of McCulloch and Pitts (1943), 
which, curiously, was not cited by Turing. However, the differences between the proposed 
network models aside, Turing took one step further by developing the notion that random 
arrangements of neurons can be suitably trained via external interference in order to compute a 
specific function, in an early proposal of a supervised learning paradigm (Turing, 1968; 
Teuscher, 2001). 


With the purpose of investigating the emergence of intelligent behavior in machines, the 
possibility of which he was absolutely convinced of, Turing proposed the study of structures 
built from certain kind of standard components that are largely random in their construction, 
which are termed unorganized machines (networks). In particular, Turing conceived two types of 
unorganized machines, referred to as A-type and B-type, and also a modified version of the B- 
type architecture, which received from Teuscher (2001) the name of BI-type. A peculiar feature 
of this last structure is the fact that, due to the presence of interference signals, an external agent 
(teacher) is capable of modifying the operation mode of each connection within the network 
aiming to adapt it to perform a desired function. 


After this preamble, we describe, in the following, the neural network models proposed 
by Turing. 


A-type Networks 


The first — and also the most simple — unorganized machine introduced by Turing is made up 
from a sufficiently high number of interconnected processing units (neurons). Each neuron: (i) 
can assume one of two possible states (O or 1) at each time instant; (ii) is connected to a central 
synchronizing unit from which synchronizing pulses are emitted at equal intervals of time (7); 
(iii) has exactly two input terminals, x,(¢)and x,(t), and produces a single output y(t) according 
to the following expression: 


yt +) =1-x (x, (1) 


Hence, each neuron computes the logical function NAND between its input signals, as 
shown in Table 1. 
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1 0 1 
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Table 1. Update of the neuron state as a function of the inputs. 


Turing’s option for the use of a NAND function as the basic unit within unorganized 
machines was probably motivated by the fact that any logical function can be implemented 
exclusively using NAND elements (Ercegovac, Lang, & Moreno, 1998; Teuscher, 2001). 
Consequently, it is possible to affirm that universal Turing machines with a truncated tape can be 
simulated with the aid of A-type machines (Teuscher, 2001). 


However, in order that an A-type network present a specific behavior, it is necessary to 
carefully design its topology and to choose the number of neurons, as well as their initial states, 
which can be an arduous task. Additionally, it is important to mention that A-type machines do 
not allow external parameter update during their operation. Figure 1 depicts a generic A-type 
unorganized machine containing five neurons. 


Figure 1. A-type network. 


B-type Networks 


The distinctive characteristic of B-type networks is the fact that each neural connection has an 
interference node that presents different operation modes. This offers the possibility of 
reinforcing useful connections and of eliminating the useless ones. Each interference node 
consists of a small A-type network containing three neurons that, depending on the initial states, 
may: (i) invert the input signal; (ii) interrupt the transmission of any information, forcing the 
output to “1” or (iii) alternately behave as described in (i) and (ii). In Figure 2, we display the 
representation adopted by Turing for a B-type connection, as well as the A-type network that 
constitutes the interference node, whereas Figure 3 exemplifies a B-type network. 
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Figure 2. B-type connection. 


Figure 3. B-type network. 


Apparently, B-type networks do not present significant modifications with respect to A- 
type machines. In fact, since the B-type link is, in essence, an A-type network, it is correct to 
state that all B-type networks are also A-type networks. However, Turing’s motivation was to 
proceed towards more flexible structures that allowed some kind of training / adaptation to 
perform specific tasks. 


In this context, B-type networks can be seen as an intermediate step: on the one hand, the 
organization of the connections still needs to be performed at the moment the network is 
conceived; on the other hand, it is possible to modify the network behavior by acting directly on 
the connections among the existing neurons. 


Bl-type networks 


Finally, Turing proposed a modification in the B-type connection structure through the insertion 
of two interference inputs i, andi, , as shown in Figure 4. 
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Figure 4. BI-type connection. 


This new class of unorganized machines, called BI-type networks by Teuscher (2001) 
(T stands for interference), offers the interesting perspective of adjusting the operation mode of 
each interference node by choosing the values of external input signals. Thus, the intention 
underlying the proposal of B-type networks becomes clearer: Turing’s purpose was to construct 
a kind of switch formed by the same basic processing element - i.e. NAND neurons -, allowing 
for an external agent (teacher) to adapt its operation mode in order that the whole network 
achieve a desired behavior. 


According to Turing, “the cortex of an infant is an unorganised machine, which can be 
organised by suitable interfering training” (Turing, 1968). In this sense, BI-type networks 
engender the process of education (training) that, according to Turing, was the responsible for 
organizing the human brain structure and enabling the learning of desired tasks, since these 
structures bring interference inputs that can be accessed and modified by an external agent. 


Discussion 


In spite of their simplicity in terms of architecture and formulation, the unorganized machines 
proposed by Turing are capable of producing very complex behaviors. Indeed, as remarked in 
Section 2.1, A-type networks can be used to implement any logical function (Teuscher, 2001). 
Turing also believed that, as occurs with A-type, B-type and BI-type networks could reproduce 
any logical function, thus being capable of implementing a universal machine with a truncated 
tape: “In particular with a B-type unorganised machine with sufficient units one can find initial 
conditions which will make it into a universal machine with a given storage capacity.” (Turing, 
1968). 


Notwithstanding, as evinced in (Copeland & Proudfoot, 1996) and (Teuscher & Sanchez, 
2001; Teuscher, 2001), this claim is not correct. In fact, there are simple logical operations that 
cannot be implemented via B-type networks, such as the exclusive disjunction (XOR) of a pair of 
inputs. 


Fortunately, with simple modifications in the B-type link, it is possible to solve this 
problem. As discussed in (Copeland & Proudfoot, 1996), Turing’s claim concerning the 
computational power of B-type networks becomes true if the available operation modes of the 
interference node are interrupt and pass, instead of the original invert (also called interchange) 
mode. Then, the authors suggested the addition of another interference node in each connection, 
so that the new link can pass through the input signal or interrupt the transmission. Figure 5 
illustrates the corresponding connection model. 


Figure 5. Copeland’s and Proudfoot’s proposal. 


A similar solution has been introduced by Teuscher (2001): to invert the input signal before 
reaching the interference node. This was accomplished by inserting another A-type neuron in the 
neural connection, as shown in Figure 6, which is functionally equivalent to the previous 
proposal, but simpler. 
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Figure 6a. TB-type connection. Figure 6b. TBI-type connection. 


Last, but not least, Intelligent Machinery brought another innovative contribution: aiming 
to exemplify the organization process of an unorganized machine, Turing defined the so-called 
P-type machines, which can be seen as universal machines without a tape, and whose description 
is initially incomplete and random. These structures were then subject to pain and pleasure 
stimuli according to the decisions made and their description was progressively adapted based on 
the reward. In Turing’s simple experiments, it is possible to notice an interesting conceptual link 
with the idea of reinforcement learning (Sutton & Barto, 1998). 


Undoubtedly, Intelligent Machinery represents a breakthrough in the study of machine 
learning, discussing and developing important concepts in the field of artificial intelligence, 
neuroscience and philosophy of mind that were well ahead of its time. We will illustrate this 
assertion following a course different from that of the crucial efforts of Copeland, Proudfoot, 
Teuscher and others, in that our emphasis will be on the points of contact between Turing’s ideas 
and modern unorganized connectionist approaches — particularly reservoir computing (RC) and 
extreme learning machines (ELMs). The fundamentals of these two approaches will be discussed 
in the following. 


3. RESERVOIR COMPUTING 


Recurrent neural networks (RNNs) can be regarded as powerful computational tools for solving 
complex and dynamic problems by virtue of: 1) the presence of feedback connections, which 
enables the development of an internal memory that captures the time history of the input 
signals; 2) the nonlinear character of the processing elements, which offers flexibility of creating 
a vast range of nonlinear mappings. In fact, these structures present the universal approximation 
property, as shown in (Funahashi & Nakamura, 1993), and reveal important conceptual 
associations with the idea of universal Turing machines (Kilian & Siegelmann, 1996). 


The effective use of RNNs nonetheless requires the adaptation of all connection weights, 
including those associated with the recurrent links, which, within a supervised learning 
framework, brings certain well-known difficulties. For instance, calculating the gradient of the 
error function becomes a computationally expensive task due to the feedback loops. Other 
common obstacles involve slow convergence rates and instability of both the training process 
and the network operation. 


In this context, echo state networks (ESNs), proposed by Jaeger (2001), and liquid state 
machines (LSMs), proposed by Maass, Natschlager and Markram (2002), offer a creative 
solution to this dilemma and establish a new recurrent neural network training paradigm known 
as reservoir computing (RC) (Verstraeten, Schrauwen, D'Haene, & Stroodbandt, 2007; 
Lukosevicius & Jaeger, 2009). 


The standard RC architecture consists of: (i) a highly interconnected and recurrent 
network of nonlinear processing elements, called dynamical reservoir (or “liquid”, in the usual 
LSM nomenclature), which is capable of preserving information about the history of the input 
signals; (ii) an adaptive linear readout, which is responsible for combining the reservoir signals 
in order to approximate the desired outputs. 


The attractive feature of the RC approaches lies in the possibility of preserving, to a 
certain extent, the processing capability inherent to a recurrent structure without incurring in the 
aforementioned drawbacks associated with the RNN training process. These objectives are 
elegantly achieved by defining the reservoir connection weights without resorting to any 
information concerning the desired task, fixing their values according to some predefined 
criterion. Hence, the overall training process consists in adapting the coefficients of the linear 
readout, which can be performed with the aid of linear regression methods (Lukosevicius & 
Jaeger, 2009). 


Figure 7 exhibits the basic RC architecture, as well as highlights the fact that the 
adaptation process employing an error signal is confined to the free parameters of the output 
layer. 
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Figure 7. RC architecture and training process. 


Echo state networks 


Consider a discrete-time RNN structure, as displayed in Figure 7, with K inputs, N internal units 
and L outputs. The reservoir input stimuli are grouped in vector u(n) =[u (n) ... u,(n)]’, 
whereas the reservoir state is represented by x(n)=[x,(m) ... Xy (n)ļ and is updated 
according to the following expression: 


x(nt+) = fW"u(nt+1l+Wx(n)+Ww”“y(n)), (2) 


where W”, W and W”“ denote the input, the internal recurrent and the output feedback 
connection weights, respectively, and f()=(f,() ... fy) specifies the activation functions 


of the internal units. The vector y(n) =[y,(”) ... y, (n)]’ contains the network outputs, which 


are determined by means of linear combinations of the reservoir states, as shown in the following 
expression: 


y(n+l) =W% x(n+1), (3) 


where W °“ is the output weight matrix. 


In his seminal work, Jaeger conceived an important conceptual element — the echo state 
property — which establishes that the reservoir states tend to reflect the recent history of the input 
signals as long as the reservoir weight matrix W satisfies specific spectral conditions (Jaeger, 
2001). This useful property allows the process of designing the ESN reservoir to be performed 
independently and previously to the network training, since the emergence of an effective 
memory of the input history is ensured, so that the weights of the recurrent connections can be 
arbitrarily assigned and remain fixed. Consequently, only the readout parameters are effectively 


trained with the aid of an error signal, and, due to its linear character, this task essentially 
involves a least-squares solution. 


However, in order that the network perform satisfactorily, the reservoir should offer a 
sufficiently rich and diverse repertoire of dynamical behaviors to the adaptive readout. Having 
these aspects in mind, Jaeger proposed a simple strategy: to create a random and sparse weight 
matrix W (Jaeger, 2001). The sparse nature of such matrix aims to decouple groups of neurons, 
contributing to the emergence of individual dynamical behaviors. 


Since their proposal, echo state networks have been applied in a myriad of important 
problems, such as system identification (Jaeger, 2003), time series prediction (Jaeger & Hass, 
2004; Sacchi, Ozturk, Principe, Cameiro & da Silva, 2007; Verplancke, et al., 2010), and channel 
equalization (Boccato, Lopes, Attux, & Von Zuben, 2011), and the obtained results were very 
promising. Nonetheless, the research field is still in its infancy, and different possibilities to 
improve the original paradigms are being constantly investigated. In particular, several works 
were devoted to the study and development of different schemes for the reservoir design which, 
in a certain sense, seek to increase the diversity and the quality of the dynamical behaviors 
originated in the recurrent layer, like (Ozturk, Xu, & Principe, 2007; Schrauwen, Wardermann, 
Verstraeten, Steil, & Stroodbandt, 2008; Boedecker, Obst, Mayer, & Asada, 2009). In addition to 
this perspective, other works, such as (Butcher, Verstraeten, Schrauwen, Day, & Haycock, 2010; 
Boccato, Lopes, Attux, & Von Zuben, 2011; Boccato, Lopes, Attux, & Von Zuben, 2012), have 
focused on proposing new readout structures with the purpose of achieving a more effective use 
of the signals coming from the reservoir. These, along with practical applications, are essentially 
the mainstream research lines involving echo state networks. 


Liquid state machines 


Independently of and simultaneously to the development of ESNs, a similar reservoir computing 
paradigm called Liquid State Machine (LSM) has emerged within a conceptual framework closer 
to neuroscience, aiming to study some computational abilities of the brain. This architecture, as 
originally proposed by Maass, Natschlager and Markram (2002), relies on a computational 
model that maps input time functions (given by streams of numbers or bits) in output streams 
(Maass, 2007) - which is done by a special filter with fading memory - and a readout stage as 
previously shown in Figure 7. 


The filter is related to dynamical systems that define the reservoir and the term “liquid” is 
justified by the dynamical patterns associated with the reservoir response to an external input. In 
contrast with ESNs, a LSM has its reservoir based on more realistic models of neurons, which 
provides a promising basis for generating a multidimensional higher-order state space for 
suitably performing generic computational tasks. Naturally, the computational power of a LSM 
will depend on how the liquid properties match the computational problem to be solved, 
something that is proved in rigorous mathematical terms in (Maass, Natschlaéger, & Markram, 
2002), giving rise to what Maass (2007) calls separation and approximation properties for LSMs. 
In consonance with the rationale of reservoir computing, the “liquid” also exhibits a sparse 
connection between neurons (Maass, Natschlager, & Markram, 2004) that evokes Turing’s 


connectionist formulations as described in Section 2. Moreover, the LSM paradigm establishes 
an important connection between unorganized network theory and some structural aspects of 
biological networks, as those identified by Yamazaki and Tanaka (2007) in the context of 
cerebellum organization and computational properties. 


4. EXTREME LEARNING MACHINES 


Feedforward neural networks (FNNs) have been extensively used in the context of information 
processing, especially because of their attractive capability of universal approximation of 
nonlinear input-output mappings (Haykin, 1998). As occurs with RNNs, it is necessary to adapt 
all connection weights in order that the network present an adequate behavior, which has been 
typically carried out with the aid of learning algorithms based on classical nonlinear optimization 
techniques (Luenberger, 2003). 


Although the absence of feedback connections simplifies the computation of the 
derivatives of the error function, there still remain certain difficulties, especially for practical 
real-time applications, due to slow convergence and costly hardware implementation of the usual 
training algorithms. In this context, extreme learning machines (ELMs) (Huang, Zhu, & Siew, 
2004; Huang, Zhu, & Siew, 2006) represent an interesting alternative to circumvent these 
limitations. The proposed solution to accelerate the training process of single-hidden layer 
feedforward neural networks consists in randomly assigning the parameters — connection weights 
and bias — of the hidden neurons, which means that only the coefficients of the linear output 
layer are effectively adapted according to an error signal. 


By relinquishing the adaptation of the hidden layer parameters, the network training 
process is converted into finding the optimal coefficients of the output linear combiner, which 
can be easily solved with the aid of linear regression methods. Hence, ELMs avoid the 
backpropagation of an error signal and the use of iterative methods (Haykin, 1998), as well as the 
possibility of convergence to local minima, which represents an improvement in terms of 
training simplicity and efficiency at the expense of a loss of structural generality. 


This creative approach receives important theoretical support from a rigorous proof 
which demonstrates that, for FNNs with a single hidden layer, under certain conditions, it is not 
imperative to adapt the intermediate neuron parameters if the activation functions of the hidden 
nodes are infinitely differentiable (Huang, Zhu, & Siew, 2006). Moreover, there are interesting 
evidences that associate the use of the Moore-Penrose generalized inverse to compute the least- 
squares solution of the output weights with a potential improvement in the network 
generalization capability (Bartlett, 1998; Huang, Zhu, & Siew, 2006). 


It is possible to notice the strong resemblance between the key ideas underlying ELMs 
and RC approaches: in both cases, the internal neural structures — the dynamical reservoir and the 
hidden layer - are not subject to adaptation, being the training process strictly focused on the 
linear combiner at the output. Indeed, we may even consider ELMs as a feedforward counterpart 
to echo state networks. As a consequence, the mathematical description of ELMs is quite similar 


to that of ESNs, and Equations (2) and (3) remain valid, with the caveat that W and W”““ must 


be set to zero as ELMs do not possess recurrent connections. Figure 8 displays the basic ELM 
architecture. 
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Figure 8. ELM architecture and training process. 
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The impressive results obtained with ELMs in applications involving time series 
prediction and pattern classification have encouraged further investigations of this new approach, 
and recent efforts led to the development of extended versions of ELMs, such as online 
sequential ELMs (Liang, Huang, Saratchandran, & Sundararajan, 2006; Huang, Wang, & Lan, 
2011), fully complex ELMs (Li, Huang, Saratchandran, & Sundararajan, 2005; Huang, Li, Chen, 
& Siew, 2008; Huang, Wang, & Lan, 2011), incremental ELMs (Huang, Chen, & Siew, 2006; 
Huang & Chen, 2007; Huang & Chen, 2008; Huang, Wang, & Lan, 2011) and kernel-based 
ELMs (Frénay & Verleysen, 2011; Huang, Zhou, Ding, & Zhang, 2012). For more details on 
state-of-art ELMs, we refer the reader to the survey of G.-B. Huang et al. (2011). 


5. REFLECTIONS ON THE POINTS OF CONTACT BETWEEN TURING’S 
NETWORKS AND MODERN UNORGANIZED MACHINES 


The exposition provided in Sections 3 and 4 revealed the conceptual and motivational 
similarities between reservoir computing approaches and extreme learning machines. Both 
proposals are characterized by the use of untrained layers of neurons whose activations are 
combined by a linear readout, which leads to significant advantages in terms of tractability and 
simplicity of the adaptation process. 


Interestingly, these approaches also converge in the sense that they evoke the idea of 
“disorganization”, a biological aspect that was taken into account in early works about neural 
networks, like (Rosenblatt, 1958) and, as will be discussed in more detail, (Turing, 1968), under 
the guise of structures like random reservoir / hidden layer. In fact, since the reservoir (liquid or 
hidden layer), is not subject to training, there will always be a part of the RC and ELM structures 
that remains essentially unorganized. 


We are now in an adequate position to present in a systematic way that which we 
consider to be the main contribution of this paper: an analysis of the points of contact between 
Turing’s pioneering views on connectionism and the modus operandi of modern unorganized 
machines. We use the expression “in a systematic way” because we believe that the main aspects 
of this analysis have been outlined in previous sections, which faces us essentially with the task 
of producing a synthesis. 


The most important aspect to be highlighted is the noticeable similarity between the 
structure of Turing’s Boolean networks and reservoir computing methods. This similarity is of a 
twofold nature, referring to the existence of a recurrent layer with neurons whose connection 
pattern is not defined in a supervised fashion that can be complemented by specific elements 
devised to incorporate the information brought by a reference signal / response. 


It should be kept in mind that, in spite of this similarity, the motivation underlying the 
two sets of proposals is fundamentally different. Turing, as discussed in Section 2, reached an 
unorganized architecture by reflecting on the process of development of the human nervous 
system from embryogenesis to children education. On the other hand, the RC and ELM 
paradigms are derived from considerations of a more practical nature, which arise from viewing 
neural networks as function / dynamical system approximators and seeking a good compromise 
insofar as the training process is concerned. It is important to mention, by the way, that the latter 
standpoint has been dominant in the field of neural networks as a consequence of the renaissance 
of the field in the 1980s, which naturally led to a certain emphasis on optimization, model 
selection and statistics. Turing’s work, in this context, becomes crucial, as it provides the neural 
network research program with an opportunity to “come full circle” and “harmonize” itself with 
the natural computing standpoint from which it arose. 


As for the need for converting the information processed by the unorganized network of 
neurons, it took different shapes in Turing’s work and in modern machines. This is due to the 
nature of the involved signals: Turing dealt with Boolean sequences, whereas RC and ELMs can 
rely on classical least squares methods that suit quite well real (and also complex) numbers. It 
should also be kept in mind that we live in a world in which computational resources are vastly 
available, which offers possibilities of developing, with the aid of simulations, stochastic training 
approaches that Turing never had the chance of testing. This makes his contribution more 
impressive, as the fundamental duality between disorganization and supervision was completely 
fathomed by him, and, moreover, his solutions to deal with it were quite ingenious. 


A point for potential cross-fertilization seems to be the perspective of modifying Turing’s 
network to generate a Boolean version of an echo state network, and, perhaps, to find a potential 
equivalent to the echo state property (Jaeger, 2001). This could be of immense value to the 
simplification of system modeling under finite alphabets, which typically gives rise to 
combinatorial optimization problems. It would also be a relevant contribution to signal 
processing in the context of finite fields (Gutch, Gruber, Yeredor, & Theis, 2012), a research 
field that may bring important contributions to applications ranging from channel coding to 
genomic data analysis. 


In addition to what has been discussed so far, it is important to remark that certain recent 
achievements of complex system theory and neuroscience also present interesting similarities 
with the essential aspects of Turing’s connectionist proposal. For instance, the recent 
transposition of graph theory to the anatomic study of cerebral connectivity (Scannell, Burns, 


Hilgetag, O'Neil, & Young, 1999) has highlighted features like: 1) segregated information 
processing regions working in parallel; 2) a dense connective structure between neurons in each 
segregated processing region; 3) a relative low number of sparse connections between different 
dedicated processing regions in order to build a distributed processing apparatus. These features 
might indicate the possibility that groups of neurons connected according to complex patterns be, 
in a certain sense, responsible for delivering “pre-processed” information to a more controlled 
layer responsible for higher level data fusion / processing, evoking the basic structures discussed 
in Sections 2 to 4. 


It is our genuine belief that the fact that Turing’s views, built in the early infancy of 
neural networks, show such intimate connections with the fundamental aspects of modern 
unorganized machines is a tribute to his ability to detect the essential factors underlying 
information processing, as well as a clear invitation to neural network researches to revisit with a 
fresh spirit the legacy of “founding fathers” like him. 


6. CONCLUSIONS 


In this work, we presented a review of Turing’s ideas on connectionist computing devices, and 
also discussed the basic elements of modern unorganized machines, like reservoir computing 
methodologies and extreme learning machines. During this exposition and in an specific section, 
we attempted to highlight the many conceptual points of contact between these two research 
programs, having in view two aims: (i) to make a contribution, in the centenary of his birth, to a 
more precise evaluation of Alan Turing’s profound views on computational intelligence; (ii) to 
indicate concrete possibilities of original developments brought by these views, which exhale, 
more than sixty years later, the unmistakable fraicheur of a brilliant mind that deserves to be 
ranked among the most original mankind has ever produced. 


Perhaps it is suitable — if not inevitable — to conclude this work with a paraphrase of the 
words attributed to the Marquis de Laplace: “Lisez Turing, lisez Turing, c’est notre maitre a 
tous.” 
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